Skip to content

fix(pipeline): link the byte-stream capsfilter after h264parse, not before - #36

Merged
Developer1010x merged 1 commit into
masterfrom
fix/airplay-macos-encoder-link
Aug 26, 2026
Merged

fix(pipeline): link the byte-stream capsfilter after h264parse, not before#36
Developer1010x merged 1 commit into
masterfrom
fix/airplay-macos-encoder-link

Conversation

@snadahalli

Copy link
Copy Markdown
Owner

The bug

Starting an AirPlay cast on macOS fails immediately, before any network traffic:

INFO  Desktop capture session ready (native GStreamer source) width=1920 height=1080
INFO  Selected encoder encoder="vtenc_h264" label="VideoToolbox H.264 (Hardware)" hw=true
ERROR AirPlay casting failed
      e=GStreamer error: Failed to link elements 'vtenc_h264-1' and 'capsfilter1'

Capture and encoder probing both succeed. The pipeline itself will not assemble.

Why

vtenc_h264 advertises stream-format: avc on its src pad as a fixed string, not a list — it cannot emit byte-stream at all. The AirPlay chain demanded byte-stream directly from the encoder:

encoder → capsfilter(byte-stream) → h264parse → appsink
          ^^^^^^^^^^^^^^^^^^^^^^^ empty intersection with vtenc_h264

h264parse is the element that converts avc to byte-stream, so it has to come first. x264enc offers { avc, byte-stream }, which is why the identical chain links on Linux and why this survived — and the Miracast pipeline already uses the correct shape (encoder → h264parse → mpegtsmux), which is why Miracast works.

Reduced, the old order reproduces the failure exactly and the new one does not:

$ gst-launch-1.0 videotestsrc ! vtenc_h264 \
    ! 'video/x-h264,profile=high,stream-format=byte-stream' ! h264parse ! fakesink
WARNING: erroneous pipeline: could not link vtenc_h264-0 to h264parse0,
  vtenc_h264-0 can't handle caps video/x-h264, profile=high, stream-format=byte-stream

$ gst-launch-1.0 videotestsrc ! vtenc_h264 ! h264parse \
    ! 'video/x-h264,stream-format=byte-stream,alignment=au' ! fakesink
(clean)

The fix

Move the capsfilter after h264parse, and drop profile=highvtenc_h264 has no profile property and does not advertise the field, so pinning it would just move the negotiation failure one element downstream.

sender_pipeline.rs (the WebRTC path) carried the same over-constraint and is fixed alongside. It is latent rather than observed there, since that path has no callers. It needs no h264parse: rtph264pay accepts avc and byte-stream alike, so constraining alignment only lets the two negotiate.

Verification

fmt --check clean, clippy --all-targets --all-features -D warnings clean, cargo test --all 207 passed / 0 failed on macOS arm64, plus the reduced gst-launch-1.0 cases above.

Not verified end to end: completing a cast additionally needs Screen Recording permission and a receiver that accepts the connection (see #27), neither of which this PR touches. What is verified is that the pipeline now links.

…efore

Starting an AirPlay cast on macOS failed immediately, before any network
traffic:

    Selected encoder "vtenc_h264" VideoToolbox H.264 (Hardware) hw=true
    ERROR AirPlay casting failed
      e=GStreamer error: Failed to link elements 'vtenc_h264-1' and 'capsfilter1'

Capture and encoder probing both worked; the pipeline would not assemble.
`vtenc_h264` advertises `stream-format: avc` on its src pad as a fixed
string, not a list, so it cannot emit byte-stream at all. The AirPlay
chain demanded byte-stream directly from the encoder:

    encoder -> capsfilter(byte-stream) -> h264parse -> appsink

leaving an empty caps intersection. h264parse is the element that converts
avc to byte-stream, so it has to come first. `x264enc` offers
`{ avc, byte-stream }`, which is why the same chain links on Linux and why
this went unnoticed — the Miracast pipeline already uses the correct shape
(`encoder -> h264parse -> mpegtsmux`), which is why Miracast works.

Reduced, the old order reproduces the failure exactly and the new one does
not:

    $ gst-launch-1.0 videotestsrc ! vtenc_h264 \
        ! 'video/x-h264,profile=high,stream-format=byte-stream' ! h264parse ! fakesink
    WARNING: erroneous pipeline: could not link vtenc_h264-0 to h264parse0,
      vtenc_h264-0 can't handle caps video/x-h264, profile=high, stream-format=byte-stream

    $ gst-launch-1.0 videotestsrc ! vtenc_h264 ! h264parse \
        ! 'video/x-h264,stream-format=byte-stream,alignment=au' ! fakesink
    (clean)

Also drops the `profile=high` constraint. `vtenc_h264` has no profile
property and does not advertise the field, so pinning it would just move
the negotiation failure one element downstream.

`sender_pipeline.rs` (the WebRTC path) carried the same over-constraint and
is fixed alongside, though it has no callers so this is latent rather than
observed. It needs no h264parse: `rtph264pay` accepts avc and byte-stream
alike, so constraining alignment only lets the two negotiate.
@snadahalli
snadahalli force-pushed the fix/airplay-macos-encoder-link branch from 435530a to 557137a Compare August 26, 2026 09:38
@Developer1010x
Developer1010x merged commit 96ed4e1 into master Aug 26, 2026
5 checks passed
Developer1010x pushed a commit that referenced this pull request Aug 26, 2026
…43)

#42 was merged twelve seconds after #41, into `fix/airplay-hkp-header` —
which #41 had just merged into master and left behind. The merge succeeded,
so nothing looked wrong, but the commits landed on a branch nothing points at
and master never received them.

Master therefore has the `X-Apple-HKP` header from #41 and none of what it was
a prerequisite for: the SRP proof still hashes g padded, transient pairing
still runs M5/M6 and gets the connection closed, and there is no encrypted
control channel. Pairing is broken on master in exactly the way #42 fixed.

This restores #42's own diff — the eight `openplay-airplay` files it actually
touched — on top of current master.

Deliberately *not* a merge of `fix/airplay-hkp-header`. That branch was cut
before #35, #36, #37, #39 and #40 merged, so a diff against it reads as
deleting `openplay-discovery/src/address.rs` and reverting 244 lines of
`openplay-sender/src/app.rs`. Merging it would silently undo five landed
fixes. Only the range between #42 and its own parent is safe to replay, and
that range touches nothing outside `openplay-airplay`.

Verified after the replay: 233 tests pass, clippy and fmt clean, and the work
from #35/#36/#37/#40 is still in the tree.

Co-authored-by: Sandeepa Nadahalli <1698507+snadahalli@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants